MS 163149.2 

B & W Ref. No. 3797,00092 
Inventor: James Mathews 
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BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to computing devices. More particularly, the present 
invention relates to a system and method for generating an input having predefined 
characteristics for a computing system. 

2. Description of the Related Art 

Most personal computer (PC) systems provide a plurality of techniques for a user to 
enter data and commands, and to manipulate data. One exemplary technique is for a user to 
enter data and command using a keyboard. Other exemplary techniques include use of a 
mouse, a track ball, and/or a voice-actuated input device. Yet another technique involves a 
using a digitizing writing pad and a stylus. 

In the situation when a conventional computer system has a digitizing writing pad and 
stylus, and a user would like to utilize the functionality that is associated with a mouse that is 
not available through the digitizing writing pad and stylus, the user must stop using the 
digitizing writing pad and style and begin to use the mouse. For example, when a user would 
like to control the behavior of the computer system using the functionality provided by the 
left- and right-buttons of a two-button mouse, the user must stop using the digitizing writing 
pad and stylus and use the mouse, or using the stylus to select the desired function from a 
sequence of menu selections. A change from a stylus to a mouse is time consuming for the 
user because the user must refocus attention from one input device to another. Further, many 
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times switching from one input device to another can be awkward or the computer system 
may be configured so that a mouse-type input device is not connected. Selecting a desired 
function from a sequence of menu selection can be plainly inconvenient. 

Thus, what is needed is a way for a user to control the behavior of a pen-based 
computing device input data so that a pen-based computing device can emulate the fimctional 
behavior of a mouse-type computer input device. 

SUMMARY OF THE INVENTION 

The present invention provides a new class of control behaviors for pen-based 
computing devices. These behaviors are easily recognized by a computer and provide a 
convenient way for users to control or invoke certain fimctions. 

The advantages of the present invention are provided by a method for detecting an in- 
air gesture. An in-air gesture is a certain prescribed movement made by a user of a pen or 
stylus that is proximate to, but not in contact with, the display screen of a computer that 
allows pen input. An in-air gesture is easy for the system to distinguish from control 
commands of prior devices because the lack of contact between pen and display clearly 
distinguishes this class of behaviors from prior control behaviors. In-air gestures also 
provide easy and quick access to system fiinctionality in a way that is more direct than in 
existing control methods. 



MS 163149.2 

B&WRef. No. 3797.00092 
Inventor: James Mathews 

Computers that are configured without traditional keyboard and mouse input devices 

and that have relatively large displays are sometimes referred to as tablet PCs. Tablet PCs 

are typically configured so that a digitizer is combined with or overlaid upon the display of 

the tablet PC. The digitizer senses the coordinates of a pen tip as the pen is moved in contact 

with the display surface. When electromagnetic displays and pens are used, the tablet PC can 

sense proximity of the pen to the display in addition to sensing just contact between the pen 

and the display. 

The present invention also provides a way for an electromagnetic pen digitizer to 
precisely emulate the functional behavior of a two-button mouse-type computer input device. 
According to the invention, a predetermined input generated by digitizing pen and a 
digitizing writing surface is received, and a user interface having a plurality of soft buttons is 
displayed in response to the predetermined input. The plurality of soft buttons provides 
selectable functionality of a two-button mouse-type computer input device. Preferably, the 
plurality of soft buttons includes a right-button function, a shift function, a control function, 
an altemate function and a bull's-eye function. An inactivity timer is started when the user 
interface is displayed; and when a user selection is not received before a predetermined 
amount of time elapses, the user interface is hidden. Depending on a received user selection 
of at least one of the plurality of soft buttons, the user interface can continued to be displayed 
for further user selections or can be hidden. When the user selection is for the bulFs-eye 
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function, the invention sends a right-button event to an application displayed below the user 
interface. 

The present invention also provides a tablet PC having a digitizing writing sxirface 
that generates a predetermined output in response to, for example, a predetermined in-air 
gesture made with a digitizing pen, and a display that displays a user interface having a 
plurality of soft buttons in response to the predetermined output. The plvirality of soft 
buttons provides selectable fimctionality of a two-button mouse-type computer input device. 
Preferably, the plurality of soft buttons includes a right-button fimction, a shift fimction, a 
control function, an alternate function and a bull's-eye function. The digitizing writing 
surface receives a user selection of at least one of the plurality of soft buttons, and the user 
interface is hidden from view on the display when the user selection of a selected soft button 
is received. When the digitizing display receives a user selection for the bull's-eye function; 
the tablet PC sends a right-button event to an application displayed below the user interface 
on the display in response to the user selection for the bull's-eye function. An inactivity 
timer is started when the user interface is displayed on the display, and the user interface is 
hidden from view on the display when a predetermined amount of time elapses without 
receiving a user selection of at least one of the plurality of soft buttons. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

The present invention is illustrated by way of example and not limitation in the 
accompanying figures in which like reference numerals indicate similar elements and in 
which: 

Figure 1 shows a schematic diagram of a conventional general-purpose digital 
computing environment that can be used for implementing various aspects of the invention; 

Figure 2 shows a tablet and stylus computer that can be used in accordance with 
various aspects of the present invention; 

Figure 3 shows an exemplary user interface (UI) that can be used for emulating the 
functional behavior of a two-button mouse-type computer input device according to the 
present invention; 

Figure 4 is a functional block diagram showing the functional relationship of an in-air 
gesture recognizer according to the present invention with a pen digitizer and an application 
program; and 

Figure 5 shows a flow diagram for a process for detecting an in-air gesture and 
emulating the functional behavior of a two-button mouse-type computer input device 
according to the present invention. 

DETAILED DESCRIPTION 

The present invention may be more readily described with reference to Figure 1-5. 
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Figure 1 illustrates a schematic diagram of a conventional general-purpose digital computing 
environment that can be used to implement various aspects of the present invention. In 
Figure 1 , a computer 100 includes a processing unit 1 10, a system memory 120, and a system 
bus 130 that couples various system components including the system memory to processing 
unit 110. System bus 130 may be any of several types of bus structures including a memory 
bus or memory controller, a peripheral bus^ and a local bus using any of a variety of bus 
architectures. System memory 120 includes read only memory (ROM) 140 and random 
access memory (RAM) 150. 

A basic input/output system 160 (BIOS), containing the basic routines that help to 
transfer information between elements within computer 1 00, such as during start-up, is stored 
in ROM 140. The computer 100 also includes a hard disk drive 170 for reading from and 
writing to a hard disk (not shown), a magnetic disk drive 1 80 for reading from or writing to a 
removable magnetic disk 190, and an optical disk drive 191 for reading from or writing to a 
removable optical disk 1 92 such as a CD ROM or other optical media. Hard disk drive 1 70, 
magnetic disk drive 1 80, and optical disk drive 1 9 1 are connected to the system bus 1 30 by a 
hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive 
interface 1 94, respectively. The drives and then* associated computer-readable media provide 
nonvolatile storage of computer readable instructions, data structures, program modules and 
other data for personal computer 100. It will be appreciated by those skilled in the art that 
other types of computer readable media that can store data that is accessible by a computer, 

-8- 



MS 163149,2 

B & W Ref. No. 3797.00092 
Inventor: James Mathews 

such as magnetic cassettes, flash memory cards, digital video disks, Bernoulli cartridges, 
random access memories (RAMs), read only memories (ROMs), and the like, may also be 
used in the example operating environment. 

A number of program modules can be stored on hard disk drive 170, magnetic disk 
190, optical disk 192, ROM 140 or RAM 150, including an operating system 195, one or 
more application programs 196, other program modules 197, and program data 198. A user 
can enter commands and information into computer 100 through input devices such as a 
keyboard 101 and pointing device 102. Other input devices (not shown) may include a 
microphone, joystick, game pad, satellite dish, scanner or the like. These and other input 
devices are often connected to processing unit 110 through a serial port interface 106 that is 
coupled to the system bus, but may be connected by other interfaces, such as a parallel port, 
game port or a universal serial bus (USB). Further still, these devices may be coupled 
directly to system bus 130 via an appropriate interface (not shown). A monitor 107 or other 
type of display device is also connected to system bus 130 via an interface, such as a video 
adapter 108. In addition to the monitor, personal computers typically include other 
peripheral output devices (not shown), such as speakers and printers. In a preferred 
embodiment, a pen digitizer 165 and accompanying pen or stylus 166 are provided in order 
to digitally capture freehand input. Although a direct connection between pen digitizer 165 
and processing unit 1 1 0 is shown, in practice, pen digitizer 165 may be coupled to processing 
unit 1 1 0 via a serial port, parallel port or other interface and system bus 1 30 as known in the 
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art. Furthermore, although digitizer 1 65 is shown apart from monitor 1 07, it is preferred that 
the usable input area of digitizer 165 be co-extensive with the display area of monitor 107. 
Further still, digitizer 165 may be integrated in monitor 107, or may exist as a separate device 
overlaying or otherwise appended to monitor 107. 

Computer 1 00 can operate in a networked environment using logical connections to 
one or more remote computers, such as a remote computer 109. Remote computer 109 can 
be a server, a router, a network PC, a peer device or other common network node, and 
typically includes many or all of the elements described above relative to computer 100, 
although only a memory storage device 1 1 1 has been illustrated in Figure 1, The logical 
connections depicted in Figure 1 include a local area network (LAN) 1 12 and a wide area 
network (WAN) 113. Such networking environments are commonplace in offices, 
enterprise-wide computer networks, intranets and the Internet. 

When used in a LAN networking environment, computer 100 is connected to local 
network 1 12 through a network interface or adapter 1 14. When used in a WAN networking 
environment, personal computer 100 typically includes a modem 115 or other device for 
establishing a communications over wide area network 113, such as the Internet. Modem 
115, which may be internal or external, is connected to system bus 130 via the serial port 
interface 106. In a networked environment, program modules depicted relative to personal 
computer 100, or portions thereof, may be stored in a remote memory storage device. 

It will be appreciated that the network connections shown are exemplary and other 
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techniques for establishing a communications link between the computers can be used. The 
existence of any of various well-known protocols such as TCP/IP, Ethernet, FTP, HTTP and 
the like is presumed, and the system can be operated in a client-server configuration to permit 
a user to retrieve web pages from a web-based server. Any of various conventional web 
browsers can be used to display and manipulate data on web pages. 

Figure 2 illustrates a tablet PC 201 that can be used in accordance with various 
aspects of the present invention. Any or all of the features, subsystems, and functions in the 
system of Figure 1 can be included in the computer of Figure 2. Tablet PC 201 includes a 
large display surface 202, e.g., a digitizing flat panel display, preferably, a liquid crystal 
display (LCD) screen, on which a plurality of windows 203 is displayed. Using stylus 204, a 
user can select, highlight, and write on the digitizing display area. Examples of smtable 
digitizing display panels include electromagnetic pen digitizers, such as the Mutoh or 
Wacom pen digitizers. Other types of pen digitizers, e.g., optical digitizers, may also be 
used. Tablet PC 201 interprets marks made using stylus 204 in order to manipulate data, 
enter text, and execute conventional computer application tasks such as spreadsheets, word 
processing programs, and the like. 

A stylus could be equipped with buttons or other features to augment its selection 
capabilities. In one embodiment, a stylus could be implemented as a "pencil" or "pen", in 
which one end constitutes a writing portion and the other end constitutes an "eraser" end, and 
which, when moved across the display, indicates portions of the display are to be erased. 
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Other types of input devices, such as a mouse, trackball, or the like could be used. 
Additionally, a user's own finger could be used for selectnig or indicating portions of the 
displayed image on a touch-sensitive or proximity-sensitive display. Consequently, the term 
"user input device", as used herein, is intended to have a broad definition and encompasses 
many variations on well-known input devices. 

A primary aspect of the invention provides a technique for emulating the fimctional 
behavior of a two-button mouse-type computer input device with a tablet PC using a pen or 
stylus as the input device. The user interface (UI) element of the invention, described in 
fiirther detail v^th respect to Figure 3 below and referred to herein as a mouse input panel 
(MIP), allows a user to quickly access the functional behavior of a two-button mouse. The 
fimctional behavior of a two-button mouse to be emulated by the pen and the MIP includes 
positional information, and an actuation signal for each button of the two buttons (i.e., left- 
and right- button down events). Also, two keyboard keys (SHIFT and CTRL) commonly 
used for modifying the fimctional behavior of the two buttons of mouse input device can be 
actuated using the MIP of the invention. Accordingly, an application program that is 
designed to operate witii input fl-om a two-button mouse input device and, therefore, expects 
a variety of mouse events can be operated using a digitizing pen v^thout modification of the 
appHcation program. 

The MIP of the present invention can be invoked, i.e., made to appear on tiie display 
of the tablet PC, in a number of different ways. One way is for a user to press a button that 

- 12- 



MS 163149.2 

B & W Ref. No. 3797.00092 
Inventor: James Mathews 

represents the MIP on a toolbar visible on the display. This, hov^ever, requires the user's 
hand to move away from the document or application to which the user's attention is focused 
to locate and depress the button that invokes the UI element. Another approach that has 
similar shortcomings is to use a hardware button on the housing of the tablet PC. The 
preferred alternative for invoking the MIP or other UI elements, or for affecting any other 
system control or input, is the use of the gesture of the present invention. 

According to the invention, "in-air" gestures made with an electromagnetic pen 
digitizer are detected for emulating the functional behavior of a two-button mouse-type 
computer input device. When the digitizing pen is detected to be above, but not in contact 
with, a digitizing writing surface, i.e., in-air, the coordinates of the in-air movements of the 
pen are recorded. Preferably, approximately the last second of the pen movement coordinate 
stream is recorded in a buffer in a well-known manner. Each time the motion of the pen is 
detected to stop, the recorded coordinates of the pen motion are evaluated by a gesture 
recognizer for determining whether the characteristics of the recorded pen movement 
matches the characteristics of one of a plurality of predefined in-air gestures. When the 
recorded pen movement matches one of the predefined in-air gestures, an action is executed 
such as the display of user interface (UI) element, thereby presenting, for example, menu 
selections that can be selected by a user using the pen. 

Preferably, the default event that is sent to an application when the pen touches the 
writing surface is a "left button down" event. When the pen touch is preceded by a selected 
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in-air gesture, no "left button down" event is sent. Instead, the UI is displayed preferably 
near the pen tip. For example, a right-handed user can select for the UI to be displayed to the 
left of the detected pen position. Alternatively, a left-handed user can select for the UI to be 
displayed to the right of the detected pen position. 

A "spike" in-air gesture is an in-air gesture that is easily made and is not likely to be 
performed accidentally by a user. There are four preferable orientations of the spike gesture 
that are detected by the present invention. A "down spike" in-air gesture is formed by 
holding the pen above a digitizing writing surface and moving the pen south (with respect to 
the north, south, east, west directions of the digitizing writing surface) and then north back to 
the original position. An "up spike" in-air gesture is formed by holding the pen above the 
digitizing writing surface and moving the pen north and then south back to the original 
position. A "right spike" in-air gesture is formed by holding the pen above the digitizing 
writing surface and moving the pen right and then left back to the origmal position. A "left 
spike" in-air gesture is formed by holding the pen above the digitizing writing surface and 
moving the pen moving left and then right back to the original position. 

These and other in-air gestures can be pre-configured, by an application developer or 
by a user, to produce any desired input to an apphcation or operating system. One preferred 
use of an in-air gesture is to invoke the MIP. When, for example, a down spike in-air gesture 
is detected, the MIP UI element is displayed and a user can, using the MIP, select one or 
more modifiers that can be attached to the next "pen down" action. Figure 3 shows an 

- 14- 



MS 163149.2 

B&WRef. No. 3797.00092 
Inventor: James Mathews 

exemplary UI element 300 that is displayed when a down spike in-air gesture is detected. 
Figure 3 shows that the selections contained in UI element 300 displayed with a down spike 
in-air gesture are detected include RIGHT 301, CTL 302, SHIFT 303, ALT 304 and 
BULL'S-EYE 305. When RIGHT 301 is selected, the next pen touch causes a "right button 
down" event. When CTL 302 is selected, a CTL key accompanies the next pen touch. 
Similarly, when SHIFT 303 is selected, a SHIFT key accompanies the next pen touch. When 
BULL'S-EYE 305 is selected, the UI pop-up window is dismissed. 

CTL 302, SHIFT 303 and ALT 304 are additional instances of the corresponding 
modifier buttons on a keyboard. Consequently, CTL 302, SHIFT 303 and ALT 304 are 
displayed as activated or locked when a corresponding instance is in either state. As soon as 
UI element 300 becomes visible, an inactivity timer is started. When a user does not tap the 
pen anywhere within UI element 303 before the inactivity timer expires, UI 300 hides, or 
disappears from view. Preferably, the duration of the inactivity timer is about 3 seconds. 
When a user taps the pen tip in the BULL' S-EYE 305, and none of CTL 302, SHIFT 302 and 
ALT 304 is active, UI 300 sends a right button click event to the window immediately below 
BULL'S-EYE 305. UI 300 then disappears from view. When one or more of the modifiers 
CTL 302, SHIFT 303 and ALT 304 is active and a user taps in the pen tip within BULL'S- 
EYE 305, UI 300 sends a right click event plus the modifier(s) to the window immediately 
below BULL'S-EYE 305. UI 300 then hides, unless one or more of the modifiers is locked. 

When a user taps on one of CTL 302, SHIFT 303 and ALT 304 buttons of UI 300, the 
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selected button flashes, thereby indicating activation. On the next pen down on, for example, 
a soft keyboard button or any area outside MIP window, UI 300 sends the modified mouse 
message and deactivates the modifier. A user can lock a modifier on UI 300 by holding on 
the button until a UI modifier lock time-out expires. Preferably, the duration of the UI 
modifier lock time out timer is about 1 second. When a user locks a modifier, the locked 
modifier is preferably displayed having a highlighting color around the perimeter of the 
locked modifier. For example, in the situation that ALT 304 is locked, a green highlight is 
displayed around the perimeter of ALT 304. While the button is locked, the UI 300 stays 
onscreen. The lock is released when the locked modifier is again tapped on. When all locks 
are released, UI 300 will be hidden fi:om view. 

Figure 4 is a fimctional block diagram showing the fimctional relationship of an in-air 
gesture recognizer 400 with a pen digitizer 401 and an application program 402. Pen 
digitizer 401 generates positional information (i.e., coordinate information) of a pen 403 in 
the form of a coordinate stream 404. Pen digitizer 401 and pen 403 are preferably 
components that are part of a conventional electromagnetic pen digitizer system. Coordinate 
stream 404 includes information relating to whether digitizer pen 403 is in contact with a 
writing surface 405 of pen digitizer 401, or whether digitizer pen 403 is in proximity above 
writing surface 405. 

When coordinate stream 404 indicates that pen 403 is in proximity above writing 
surface 405, in-air gesture recognizer 400 records coordinate stream 404 in a buffer 406. 
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Preferably, buffer 406 records approximately the last second of the pen movement coordinate 
stream in a well-known manner so that about 200 coordinate points are recorded for 
evaluation. Alternatively, the time span of pen movement information recorded in buffer 406 
can be optimized for each user. When in-air gesture recognizer 400 detects that the motion 
of pen 403 stops, the recorded pen motion coordinate information is evaluated by a gesture 
detector 407 for determining whether the characteristics of the recorded pen movement 
match the characteristics of a predefined in-air gesture. More specifically, gesture detector 
407 compares recorded motion characteristics, such as motion shape, motion size, start and 
end points of the motion, and speed of the motion, to corresponding characteristics for a 
predefined in-air gesture, such as a down spike gesture. Of course, other in-air gestures can 
be detected using the same technique. 

When gesture detector 407 determines that the recorded pen movement matches the 
predefined in-air gesture for a down spike, a user interface (UI) element 408 is displayed that 
presents menu selections that can be selected by a user using the pen digitizer. The particular 
menu selections contained in the UI emulate the functional behavior of a two-button mouse- 
type computer input device. 

Figure 5 shows a flow diagram 500 for a process for detecting an in-air gesture and 
emulating the fimctional behavior of a two-button mouse-type computer input device 
according to the present invention. The process begins at step 501. At step 502, it is 
determined whether the pen is in proximity to and in the air above the digitizing writing 
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surface. If not, the process remains at step 502 until the pen is determined to be in proximity 
to and in the air above the digitizing writing surface, at which time flow continues to step 
503 where the coordinate information stream generated by the in-air gesture of the pen is 
recorded in buffer 406 (Figure 4). Flow continues to step 504, where it is determined 
whether the in-air pen movement has stopped. If not, flow continues to step 503, where the 
recording of the coordinate information stream generated by the in-air gesture of the pen 
continues in buffer 406. 

If, at step 504, it is determined that the in-air gesture has stopped, flow continues to 
step 505, where it is determined whether the recorded in-air gesture corresponds to, for 
example, a down spike in-air gesture. If so, flow continues to step 506, where an action 
corresponding to the detected in-air gesture is displayed. For example, when a down spike 
in-air gesture is detected, UI 300 is displayed. Altematively, a control function can be 
executed when a particular in-air gesture is detected. If, at step 505, the recorded in-air 
gesture of the pen does not correspond to a down spike gesture, flow retums to step 502. 

As an alternative to the embodiment shown in Figure 5, the digitizer pen and 
digitizing writing surface can be configured so that a user manually selects, for example, a 
button that defines the digitizing pen to be in a hovering state. For this alternative 
embodiment, the decision made in step 502 would be based on the user-selected state for the 
pen, i.e., hovering or not hovering. 

While the present invention does not rely on complex in-air gestures for emulating 

- 18 - 



MS 1631492 

B & W Ref. No. 3797.00092 
Inventor: James Mathews 

the functional behavior of a two-button mouse-type computer input device, other in-air 
gestures other than spike movements can altematively be utilized by the present invention. 
For example, other suitable in-air gestures that can be used with the present invention include 
circularly 5 triangularly, or rectangularly shaped motions, in addition to a saw tooth motion or 
a reciprocating motion. Moreover, a UI menu or control window other than a mouse-type 
input panel could be emulated by using an in-air gesture that is detected by the present 
invention. Exemplary UI menus or control windows that could be controlled by in-air 
motions detected by the present invention include a file management menu, an edit function 
menu, and a formatting menu. Further still, an in-air gesture according to the present 
invention can be used for generating specific keystrokes, such as a space, backspace and 
carriage return, or user definable keystrokes and/or sequences of keystrokes. 

While the present invention has been described in connection with the illustrated 
embodiments, it will be appreciated and understood that modifications may be made without 
departing fi-om the true spirit and scope of the invention. 
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